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Foreword 

The  problems  of  evaluating  new  curriculum  materials   such  as 
those  produced  by  UICSM  are  manifold  and  not  easy  to  solve.     A  pri- 
mary objective  which  UICSM  set  for  itself  when  it  was  organized  in 
/    1931  was  to  improve  the  learning  of  the  skills  and  facts  comprising 
the  college  preparatory  sequence  found  in  most  high  schools  at  that 
time.     In  trying  to  achieve  this  objective,    UICSM  made  major  changes 
in  pedagogy  and  major  changes  in  content  especially  with  respect  to 
logical  organization.      Teaching-by-discovery  replaced  teaching-by- 

'      telling  in  order  to  build  interest  and  to  tap  the  creative  talents  of 

students.     We  maintained  very  high  standards  of  precision  of  language 
in  our  instructional  materials.     In  order  to  achieve  the  understanding 
which  intellectual  integrity  demands  as  a  prerequisite  to  the  acquisition 

X      of  skills,    we  introduced  the  notion  of  a  deductive  organization  of  elemen- 
tary algebra.      Thus  students  derived  the  theorems  which  justified  the 
usual  manipulation  rules  and  they  became  aware  of  the  logical  connec- 
tions among  the  manipulation  rules  [e.g.,    the  rules  for  combining  like 
monomials  and  the  rule  for  combining  like  radicals  are  logical  conse- 
quences of  the  sanne  principle].     Topics  from  the  elementary  theory  of 

^       sets  were  introduced  to  give  students  insights  into  what  made  the 

mechanical  procedures  work  when  they  solved  equations  and  inequations 
or  when  they  solved  systems  of  equations  by  graphing.     Thus  UICSM 
students  had  the  opportunity  not  only  to  acquire  the  skills  and  knowledge 

expected  from  the  "traditional"  curriculum  but  also  to  learn  mathemat- 

> 

ical  ideas  and  techniques  which  were  not  provided  for  in  school 
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matheniatics  programs  prior  to  UICSM.     In  addition,  as  we  were  told 
by  cooperating  teachers  in  experinnental  classes  and  as  we  observed  on 
our  visits  to  these  classrooms,    UICSM  students  were  routinely  develop- 
ing insights  into  mathematics  and  attitudes  toward  new  problems  which 
were  acquired  by  only  the  best  students  in  the  best  of  traditional  classes 
and  which  nnathematics  educators  have  professed  to  be  among  the  most 
significant  outcomes  of  instruction. 

Any  comprehehsive  evaluation  program  should  consider  both  the  pri- 
mary skills -and-knowledge  objective  which  is  common  to  both  our  program 
and  the  traditional  one,    and  the  rather  astounding  secondary  objectives 
which  we  appeared  to  acconnplish  almost  as  by-products.     In  the  early 
years  of  our  work  we  were  able  to  look  only  at  the  primary  objective  in 
making  comparative  studies  of  UICSM  students  and  students  trained  in  the 
traditional  curriculum.     Naturally,    it  was  not  possible  to  make  connpara- 
tive  studies  with  respect  to  new  content,    and  we  had  neither  the  skill  nor 
the  time  to  develop  objective  tests  for  making  comparative  studies  with 
respect  to  the  development  of  insight  and  creativity  in  attacking  new^  prob- 
lems.     What  we  could  do  and  what  we  did  was  to  collect  data  on  students' 
achievennent  as  measured  by  standardized  algebra  and  geometry  tests 
which  were  designed  for  the  pre- 1950  traditional  curriculum.     We  do  not 
pretend  that  these  tests  have  provided  us  with  a  complete  description  of 
our  students'  achievement,    and  we  recognize  that  the  language  used  in 
many  of  the  items  on  these  tests  was  somewhat  unfamiliar  to  our  students. 
Nevertheless,   we  were  willing  to  use  the  test  results  as  a  rough  guide  in 
revising  our  instructional  materials.      The  study  reported  below  is  the 
first  we  have  published  based  on  an  analysis  of  the  data  collected  by  means 
of  such  tests. 
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The  results  of  the  analysis  are  not  as  satisfactory  as  we  might  have 
hoped.     Achievement  by  students  in  our  pilot  and  participating  schools 
was  not  uniformly  high.      These  results  are  in  accord  with  the  subjective 
reports  we  have  received  from  teachers  and  with  very  rough  analyses 
which  we  carried  out  some  years  ago  as  the  data  were  collected.      These 
preliminary  findings  persuaded  us  to  revise  our  instructional  materials 
for  our  advanced  units  in  the  direction  of  including  many  review  sections 
of  miscellaneous  manipulative  exercises.      The  present  study  will  be 
useful  to  the  extent  that  the  objectives  measured  by  the  tests  are  valid 
in  our  continuing  curriculum  development  work. 

The  report  which  follows  is  the  first  to  be  produced  by  the  newly- 
organized  research  section  of  UICSM.     It  is  directed  to  a  varied  audience-- 
research  workers  in  mathematics  education,    teachers,    administrators, 
and  others  who  have  been  interested  in  UICSM.     Such  variety  posed  a 
problem  for  the  authors.      Professors  Tatsuoka  and  Easley  have  felt 
obliged  to  include  sufficient  details  of  the  statistical  analysis  to  answer 
the  research  workers'  questions  but,    at  the  same  time,    did  not  wish  to 
overwhelm  the  nonstatistically  oriented  reader.     They  have,    consequently, 
employed  the  device  of  setting  off  the  technical  sections  with  wider  margins 
and  closer  spacing,    so  that  the  general  reader  may  skip  these  if  he  wishes. 
They  have  also  provided  an  abstract  for  those  who  wish  a  brief  resume. 

Our  research  section,    under  the  direction  of  Professor  Easley,    is 
presently  engaged  in  further  analysis  of  available  data,    in  collecting  new 
data  for  studies  underway,    and  in  designing  further  studies  related  to  the 
teaching  and  learning  of  mathematics.     These  research  activities  will  play 
an  important  role  in  the  development  of  new  curriculum  materials  which 
is  the  primary  job  of  UICSM, 


Max  Beberman 
Director,    UICSM 


Urbana,   Illinois 
September  1,    1963 
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Abstract 

This  report  describes  the  results  of  an  evaluation  study 
in\'olving  approximately    1700    UICSM  and  nearly    700    non- 
UICSM  algebra  students.      The  former  are  designated  the 
"experimental"   sample  and  the  latter  the  "control"   sample. 
The  experimental  sample  was  broken  do'wn  into  six  groups  de- 
pending on  grade  (8th  or  9th),    version  of  UICSM  First  Course 
text  used  (1958  or  1959),    and  duration  of  study  prior  to  testing. 

Achievement  in  algebra  -was  measured  by  the  Cooper- 
ative Algebra  Test  (Elementary),    Forms  T,   X,    and  Y.     The 
analysis  of  covariance  was  used  in  comparing  group  means 
on  this  test,    with  adjustments  for  inequalities  between  groups 
on  the  Differential  Aptitude  Tests  of  Numerical  Reasoning 
and  of  Verbal  Ability, 

With  these  adjustments,   the  experimental  sample  as  a 
whole  showed  significantly  greater  achievement  than  the  con- 
trol sample.      This  was  also  true  of  four  of  the  six  experi- 
mental groups  taken  separately. 

The  adjusted  Coop  Algebra  means  of  the  other  two  groups 
were  not  significantly  different  from  that  of  the  control  sample. 
Possible  reasons  for  these  results  are  discussed  in  the  report. 

Description  of  Samples  Tested  in  the  Study 

The  experimental  sample  comprised  1,705  students  in  75  eighth-  and 
ninth-grade  classes  taking  the  UICSM  First  Course  (Units  I-IV)  during  the 
1958-59  and  1959-60  school  years.     They  fall  in  the  upper  two-thirds  of 
the  college  preparatory  students  in  38  schools   scattered  throughout  the 
country.      The  samples  were  not  arrj\'ed  at  b.'  stratified  sampling  but  by 


Most  of  the  data  used  in  this  paper  were  originally  obtained  and  subjected 
to  a  different  type  of  analysis  by  O.  Robert  Brown,  Jr.  in  an  unpublished 
report  of  the   same  title. 

The  authors  are  especially  indebted  to  Judy  Boyle  for  compilini^  the  data  and 
assisting  in  the  calculations  and  to  Eleanor  McCoy  for  compiling  a  list  of 
changes  made  in  the  latest  version  of  the  UICSM  first  course  text.      We  are 
also  indebted  to  Robert  Kansky  for  a  numoer  of  helpful  comments  on  the  M5. 
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historical  accident,    from  schools  which  volunteered  to  try  the  UICSM  mate- 
rials.     (Due  to  this  fact,    it  was  necessary  to  introduce  statistical  controls, 
as  described  in  the  section  on  "Statistical  Analysis.")     This  sample  was 
divided  into  six  groups   (designated  by  E    ,    E    ,    E    ,    E    ,    E    ,    and  E,),    de- 
pending on  the  grade,    the  ochool  year  in  which  the  UICSM  First  Course  was 
taken,    and  the  time  of  testing.     Groups  E     and  E     contained  25%  of  the 
UICSM  students   starting  First  Course  in  September,    1958,    and  the  other 
four  represented'43%  of  the  students  starting  in  September,    1959. 

The  particular  year  (1958-59  or  1959-60)  in  which  the  course  was 
taken  is  important  because  the  UICSM  material  published  and  first  used 
in  the  fall  of  1959  v/as  the  result  of  a  revision  which,    it  was  hoped,    would 
improve  student  performance  on  standard  high  school  algebra  tests.     While 
earlier  versions  of  the  UICSM  texts  had  concentrated  primarily  on  making 
ideas  clear,    the  1959  revisions  reflected  the  realization  that  clarity  of  con- 
cept development  was  not  enough  to  ensure  that  concepts  are  retained  by 
students  in  a  useful  form.     Although  further  improvement  of  clarity  of  con- 
cepts was  not  avoided  in  making  these  revisions,    the  principal  purpose  for 
the  revisions  was  to  provide  increased  practice  in  using  mathematical  ideas, 
The  diversity  of  ways  in  which  opportunities  for  practice  w^ere  increased 
in  the   1959  versions  of  Units  I-III  exemplified  by  the  new  topics  and  treat- 
ments briefly  described  below. 

In  Unit  I,    the  explanation  of  and  the  exercises  on  the  grouping  con- 
ventions were  considerably  expanded,    and  the  term  "  unabbreviating"  was 
introduced  to  refer  to  the  procedure  of  putting  omitted  symbols  back  into 
an  expression.      The  treatment  of  some  of  the  principles  and  concepts 
relating  to  real  numbers  was  made  more  explicit,    and  more  exercises 
were  included  which  involved  the  use  and  recognition  of  basic  principles 
for  real  numbers.     In  Unit  II,    the  treatment  of  simiplification  problems 
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involving  division  was  considerably  expanded  and  the  total  number  of  supple- 
mentary exercises  was  approximately  doubled.     The  new  exercises  were 
mainly  concerned  with  proofs  and  simplification  problems  involving  divi- 
sion--in  contrast  to  the  old  supplementary  exercises  (still  retained)  which 
concerned  only  substitution  and  algebraic  simplification.     In  Unit  III,    the 
application  of  uniqueness  and  cancellation  principles  to  the  solution  of  linear 
equations  and  "inequations,"  quadratic  equations  and  square  roots  received 
a  more  detailed  treatment. 

Qualitatively,    teachers  and  writers  were  satisfied  that  these  changes 
would  promote  an  increased  mastery  of  algebraic  skills  by  the  pupils.     It 
is  most  interesting,   therefore,    to  inquire  whether  students  using  the  new 
edition  showed  such  an  increased  mastery,    as  measured  by  a  test  empha- 
sizing manipulative  skill. 

The  time  of  testing  for  achievement  in  the  experimental  sample  was 
rather  disuniform  from  one  class  to  another.     Some  classes  (those  in 
groups  E2  and  E .  )  were  tested  at  the  end  of  the  school  year  in  which  they 
had  started  their  UICSM  courses,    and  most  of  them  had  just  begun  their 
study  of  Unit  4  at  the  time  of  testing.     These  students  were  at  a  disadvan- 
tage in  that  the  criterion  tests  (described  later)  included  items  on  exponents 
and  on  geometrical  applications  of  algebra- -topics  which  were  not  covered 
in  Units  1-3.     Other  classes  were  tested  upon  completion  of  Unit  4,   which 
generally  took  place  some  time  during  the  fall  of  the  school  year  following 
the  one  in  which  they  had  begun  the  course.     The  distribution  of  times  of 
testing  of  the  various  classes  is  indicated  graphically  in  Figure  1,   which 
also  shows  the  size  of  the  groups  into  •which  the  experimental  and  control 
samples  were  divided. 

Note  that  the  year  and  month  in  which  each  group  began  its  algebra 
course  is  indicated  by  the  position  of  the  left  border  of  its  block  along   the 
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Figurt  I 

Block  diagram  showing  the  groups  into  which  the  control  and  experinnental 
samples  were  divided.     The  left  edge  of  each  block  indicates  the  time  at 
which  that  group  began  its  course.     The  distribution  of  times  of  testing 
for  these  groups  is  indicated  by  the  right  hand  boundaries  of  the  blocks. 
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time  scale  separating  the  blocks  representing  the  control  and  experimental 
groups.     The  right-hand  border  indicates  the  time  of  testing  with  the  Coop 
Algebra  Test.     For  those  groups  which  contained  classes  that  varied  in 
time  of  testing,    the  right-hand  border  is  made  step-wise  to  indicate  the 
variations.     The  heights  of  these  steps  are  drawn  roughly  proportional 
to  the  number  of  students  tested  in  each  time  period;   the  total  height  of 
each  block  is  proportional  to  the  group  size. 

The  breakdown  of  the  experimental  sample  into  six  groups,   in  accord- 
ance with  the  characteristics  described  above,    is  summarized  in  the  upper 
portion  of  Table  1,   below. 

Table  1.     Subdivisions  of  experimental  and  control  samples 

into  several  groups  based  on  some  relevant  features. 


Group 
^1 

Grade 
8 

Year  Course 
Was 
Begun 

Time  of  Testing 
May  '60-Mar  '61 

N 

1959 

226 

o 
8 

W 

^2 

9 

1958 

May  '59 

118 

^3 

9 

1958 

Sept  -  Dec  '59 

270 

1 

^4 

9 

1959 

May  '60 

574 

e 

^5 
^6 

9 
9 

1959 
1959 

Nov.-  Dec  '60 
Jan  '61 

382 
135 

Control 
Sample 

^1 
^2 

9 
9 

1958 
1959 

May  '59 
May  '  60 

515 
161 
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The  control  sample  comprised    676    students  in    Z6    ninth-grade  classes 
taking  "conventional"  algebra  courses  during  1958-59  and  1959-60  in  schools 
which  also  had  UICSM  classes.      The  control  classes  were  from    1 1  of  the    38 
schools  represented  in  the  experimental  sample  and    10    others.     (All  schools 
using  UICSM  text  were  encouraged  to  set  up  both  experimental  and  control 
classes,    but  many  were  unable  to  do  so  or  failed  to  administer  all  the  tests.  ) 
This  sample  was  divided  into  only  two  groups  as  shown  in  the  lower  portion 
of  Table    1,    (C^:    the  1958-59  classes;     C^:    the  1959-60  classes).     Further 
subdivision  w^as  unnecessary  because,    unlike  those  in  the  experimental 
sample,    the  control  classes  were  \iniform  with  respect  to  grade  and  time 
of  testing.      That  is,    all  control  classes  were    9th-grade  classes,    and  all 
were  tested  at  the  end  of  the  school  year  in  which  they  had  their  first 
algebra  course. 

Test  Data  Used 

The  criterion  variable  in  terms  of  which  intergroup  comparisons 
w^ere  made  w^as  the  Cooperative  Elementary  Algebra  Test  (Forms    T,    X, 
and    Y),    published  by  the  Educational  Testing  Service.      This  test  empha- 
sizes computational  skills,    and  hence  provides  an  "acid  test"  for  UICSM- 
trained  students --especially  for  those  w^ho  were  taught  with  material  prior 
to  the  1959  revision. 

The  means  and  standard  deviations  of  the  Coop  Algebra  Test  scores 
for  the  experimental  and  control  samples,    as  well  as  for  the  several  sub- 
groups,   are  shown  in  Table    2,    below.     It  is  seen  that  the  experimental 
(UICSM)  sample  as  a  whole  (and  each  subgroup  thereof)  had  a  higher  mean 
score  than  either  of  the  control  ("conventional")  subgroups.      The  superior 
performance  of  UICSM-trained  students  must,    however,    be  partly  attri- 
buted to  the  selection  factor:    they  represented  the  upper  two-thirds  of 
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Table  2.      Means  and  Standard  Deviations  of  the  Coop  Algebra  Tests, 
DAT-VR,    and  DAT-NA  for  the  experimental  and  control 
samples  and  the  several  subgroups  thereof.      (Also  shown 
are  the  coefficients' of  multiple  correlation  of  Coop 
Algebra  on  the  two  DAT  subscores.) 


TEST 


Coop  Algebra 
DAT-VR 
DAT-NA 
Multiple    r 


Total 
Experimental 


Z     3     4     5     6 


Mean        S.D.      Mean         S.D.     Mean        S.D.       Mean        S.D. 


63.37      10.19  66.89      10.03  62.83      10.10  66.58      7.94 

26.98        8.11  27.39        7.61  26.92        8.18  28.42      7.51 

24.62        7.19  23.72        7.90  24.76        6.98  26.14      5.62 

.645                          .670  .653                          .602 


TEST 

Mean 
63.72 

S.D. 
8.48 

Mean 
59.57 

S.D. 
11.22 

Mean 
65.83 

S.D. 
8.  61 

Mean        S.  D. 

Coop  Algebra 

63.18      9.50 

DAT-VR 

28.57 

7.86 

24.22 

8.31 

28.62 

7.68 

28.95      7.02 

DAT-NA 

25.69 

6.68 

22.95 

7.66 

26.04 

6.39 

25.76      5.51 

Multiple   r 

.660 

.671 

.527 

.  649 

Total 
Control 

C 

1 

C 

'2 

TEST 

Mean 
58.26 

S.D. 
11.09 

Mean 
58.77 

S.D. 
11.33 

Mean 
56.65 

S.D. 
10.  11 

Coop  Algebra 

DAT-VR 

23.63 

8.02 

23.46 

8.23 

24.  19 

7.27 

DAT-NA 

21.26 

7.20 

21.66 

7.38 

23.09 

6.48 

Multiple   r 

.673 

.718 

•  ' 

569 

The  norms  for  this  test,    based  on    15,000  students  in    130    schools,    show  a 
mean  of   52.  9   with  a  standard  deviation  of   9.  6. 
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the  college  preparatory  students  in  each  school,    as  mentioned  earlier.     On 
the  other  hand,   the  control  sample  was  formed  from  algebra  classes  for 
which  the  admission  requirements  were  generally  not  so  stringent. 

In  order  to  make  adjustments  for  the  inequality  in  "scholastic  ability" 
between  the  experimental  and  control  samples,    the  Differential  Aptitude 
Test  Battery  (DAT)  results  for  all  individuals  in  our  sample  were  employed. 
This  battery  had  been  administered  prior  to  the  students'  taking  their    alge- 
bra courses,   UICSM  or  "conventional",    as  the  case  may  be.     Specifically, 
the  scores  on  the  t-wo  subtests-- Verbal  Reasoning  (VR)  and  Numerical 
Ability  (NA)--  of  the  DAT  were  used  as  statistical  control  variables  as 
described  in  the  next  section.     The  means  and  standard  deviations  of 
DAT-VR  and  DAT-NA  for  the  various  groups  are  shown  in  Table  2.     As  to 
be  expected,    the  experimental  groups  have  higher  means  than  the  control 
groups. 

It  is  recognized  that  adjustments  for  DAT-VR  and  DAT-NA  scores 
are  probably  insufficient  to  compensate  for  all  the  relevant  but  uncontrolled 
inequalities  existing  bet^veen  experimental  and  control  samples.     For 
instance,   the  "teacher  variable"  is  quite  likely  to  be  a  significant  factor 
contributing  to  the  difference  in  algebraic  achievenaent  between  the  two 
samples.      But  the  only  measure  available  for  assessing  "teacher  compe- 
tence" was  the  very  crude  one  of  the  number  of  years  of  experience,    and 
it  was  decided  that  little  would  be  gained  by  making  adjustments  for  this 
variable.      The  adjustments  which  we  made,    therefore,    probably  repre- 
sent the  most  that  could  be  done  within  the  limits  of  available  data. 

Some  indication  of  the  extent  to  which  these  adjust- 
ments were  effective  can  be  gained  by  examining  the  magni- 
tudes of  the  coefficients  of  multiple  correlation  of  Coop  Algebra 
scores  on  the  two  DAT  subscores.     These  ranged  from  about 
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.53  to  .72,    depending  on  the  particular  group,    as  shown  in 
Table  2.      Thus,    it  can  be  argued  that  the  two  DAT  scores 
accounted  for  from    30   to    50   percent  of  the  variance  in  the 
Coop   scores. 

Statistical  Analysis 

The  statistical  technique  used  for  making  a  majority  of  the  intergroup 
comparisons  described  below  was  the  standard  analysis  of  covariance. 
This  technique  amounts  to  making  an  estimate  of  what  the  Coop  Algebra 
mean  of  each  group  under  comparison  would  have  been  if  the  DAT-VR  and 
DAT-NA  means  had  each  been  equal  for  the  several  groups,    and  then 
making  a  comparison  between  these  estimated  (or  "adjusted")  Coop  Alge- 
bra means. 

In  actual  practice,   it  is  the  sums-of-squares  of  the 
Coop  Algebra  tests  that  are  adjusted.     The  test  of  signifi- 
cance then  exactly  parallels  that  for  the  usual  analysis  of 
variance.     That  is,   the  ratio  of  the  adjusted  between-groups 
mean- square  to  the  adjusted  within-groups  mean- square  is 
used  as  an  F-ratio.     If  the  value  of  this  ratio  exceeds  a  criti- 
cal value,   the  adjusted  Coop  Algebra  means  are  judged  to  be 
significantly  different  from  each  other. 

In  this  manner,    comparisons  were  made  between  the  experimental 
sample  as  a  whole  and  the  control  sample  as  a  whole;   between  groups  E_ 
through  E/   (9th  grade  classes  in  experimental  sample)  and  the  control 
sample;    and  so  forth.     Each  comparison  was  designed  to  test  the  effect  of 
some  particular  variable  or  combination  of  variables  on  which  the  contrasted 
groups  differed  systennatically. 

Experimental  Sample  vs.    Control  Sample.    The  analysis-of-variance 
table  for  the  adjusted  sum-of-squares  (S.  S. )  in  this  comparison  is  shown 
in  Table  3,    as  are  the  adjusted  means. 
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Table  3.      Adjusted  analysis  of  variance  table  for  comparing 
experimental  group  and  control  group  sample  Coop 
Algebra  means 

Source Adjusted  S.  S.  n.  d.  f.  Adjusted  Mean-Square      F-ratio 

Between-groups  1944.83  1  1944.83  31.19 

Within-groups  148214.00  2377  62.35 

Total  2378 


Adjusted  Coop  Algebra  means 


Group  E^  ^   ,    =    62.  50 
'^      total 

Group  C^^^   ^   =    60.45  *P    <    .001 


The  resulting  F-ratio   (31.  19)  indicates  that  the  adjusted  Coop  Algebra 
means  for  the  two  samples  are  significantly  different  at  the  .001-level,   the 
direction  of  the  difference  being  in  favor  of  the  experimental  (UICSM) 
sample.     It  should  be  noted  that  the  adjustments  for  inequalities  on  the  two 
DAT  nneans  have  lowered  the  experimental  sample's  Coop  Algebra  mean 
from  63.37  (cf.    Table  2)  to  62.50,    and  raised  the  corresponding  control- 
sample  mean  from  58.  26  (cf.    Table  2)  to  60.45.     This  is  the  way  in  which 
the  analysis  of  covariance  method  operates  in  order,    at  least  partly,   to 
compensate  for  group  inequalities  on  related  variables. 

We  may  thus  conclude  that  the  UICSM  sample  as  a  whole  showed  a 
significantly  higher  algebra  achievement  (as  measured  by  the  Cooperative 
Elementary  Algebra  Test)  than  did  the  "conventional"  algebra  sample  as  a 
whole- -even  when  due  compensations  are  made  for  the  fact  that  the  former 
was  superior  to  the  latter  in  general  academic  ability  as  measured  by  the 
DAT  Verbal  Reasoning  and  Numerical  Ability  scores. 
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Group  E^  vs.    Groups  E,^  /% .     In  view  of  the  fact  (cf.    Table  2) 

that  the  8th -grade  experimental  classes  (group  E  )   scored  considerably 

higher  on  the  Coop  than  did  the  9th-grade  experimental  classes 

(groups  E,^  /v).    it  is  pertinent  to  inquire  whether  the  means  for  these 

two  sub-samples  are  significantly  different.     For  if  so,   there  arises  the 

possibility  that  the  superiority,    seen  above,    of  the  experimental  sample 

is  attributable  to  just  the  8th -grade  segment,   which  constitutes  about  13% 

of  the  experimental  sample. 

The  adjusted  analysis-of-variance  table  for  the  experimental  8th-  vs. 

9th-grade  comparison  is  shown  in  Table  4,     The  resulting  F-ratio  indicates 

a  significant  difference  in  favor  of  the  experimental  8th-grade  classes. 

Note  that,    in  this  case,   the  adjustments  have  widened 
the  gap  between  the  group  means  under  comparison.     Refer- 
ence to  Table  2  shows  that  the  8th  graders  had  a  lower  DAT- 
NA  mean  than  the  9th  graders,    although  the  DAT-VR  mean  of 
the  former  was  higher  than  that  of  the  latter.     Hence,    the 
adjustments  for  the  two  predictors  worked  in  opposite  dir- 
ections in  this  case,   and  the  fact  that  the  8th-grade  mean  was 
raised  from  66.  89  to  67.  33  while  that  of  the  9th  graders  was 
lowered  from  62.  83  to  62.  76  indicates  the  overriding  effect 
of  the  compensation  for  unequal  Numerical  Ability  scores. 

At  any  rate,    since  the  8th -grade  classes  were  found  to  be  signifi- 
cantly superior  to  the  9th -grade  experimental  classes  on  the  Coop  Algebra 
Test,  it  is  desirable  to  inquire  whether  or  not  the  superiority  of  the  experi- 
mental to  the  control  sample  will  continue  to  hold  when  the  8th-grade 
group  is  removed  from  the  experimental  sample. 

Experimental  Groups  E.,  a\  '^^'    Control  Sample.     That  the 

answer  to  the  question  posed  above  is  in  the  affirmative  can  be  seen  from 
the  results  presented  in  Table  5.     The  experimental  sample  shows  a 
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significantly  higher  overall  performance  on  the  Coop  Algebra  Test  than 
the  control  sample,    even  after  the  especially  high- scoring  experimental 
group  (the  8th-grade  classes)  is  removed  from  consideration. 

Table  4.      Adjusted  analysis  of  variance  table  for  comparing  Group  E^ 
and  Groups  E,.,  ,,   means  on  Coop  Algebra 

Source Adjusted  S.  S.  n.  d.  f .  Adjusted  Mean-Square      F-ratio 

Between-groups  2027.47  1                        2027.47                        34.79" 

Within-groups  99.082.24  1701                            58.25 

Total  1702 


Group  E.  =    67.  33  ^ 

Adjusted  Coop  Algebra  means  P    <    ,001 

Group  E,^  />  =    62.76 

(2.  ....  b) 


Table  5.      Adjusted  analysis  of  variance  table  for  comparing 

experimental  groups  E.^  /,   and  control -sample 

means  on  Coop  Algebra 

Source Adjusted  S.  S.       n.  d.  f .       Adjusted  Mean-Square      F-ratio 

911.21  14.90* 

61.  14 


Between-groups 

911.21 

1 

Within-groups 

131. 504. 11 

2151 

Total 

2152 

* 


Groups  E.p  /.    =    61.85 

Adjusted  Coop  Algebra  means  P    <    .  001 

Group  C  J  =60.04 
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Individual  Experimental  Groups  vs.    Control  Sample.     We  now  com 
pare  each  of  the  experimental  9th-grade  groups  (E,,    E^,    E.    and  E_),    in 
turn,   with  the  control  sample.     Group  E,  was  not  considered  because  it 
was  both  very  small  in  size  (N   -    135)  and  extreme  in  its  delayed  testing 
time,    as  seen  in  Table  1.     The  results  of  the  four  comparisons  are  sum- 
marized in  Table  6,    showing  that  two  of  the  experimental  groups  (E-,  and  E^^) 
were  found  to  be  significantly  superior  to  the  control  sample  while  two 
(E-  and  E.  )  were  not  significantly  different  from  it. 

It  should  be  pointed  out  that,    for  the  connparison  invol- 
ving group  E_,    an  unrestricted  linear  hypothesis  model 
(using  separate  group  regression  weights)  was  used  instead 
of  the  standard  covariance  analysis.        This  is  because  the 
equality-of-slopes  condition,   prerequisite  to  regular  covari- 
ance analysis,   was  not  satisfied  in  this  case. 

The  two  experimental  groups  that  failed  to  achieve  adjusted  Coop 
Algebra  means  which  were  significantly  higher  than  that  of  the  control 
sample  apparently  did  so  for  two  different  reasons.     Group  E^  classes, 
having  taken  their  UICSM  course  prior  to  the  1959  revision  of  materials, 
•were  at  a  disadvantage  as  noted  earlier,   in  not  having  sufficient  drill  in 
the  manipulative  skills.     Group  E.,    on  the  other  hand,   was  tested  before 
completion  of  Unit  4,    and  hence  had  not  yet  covered  some  of  the  topics 
that  were  included  in  the  Coop  test. 

Group  Ej.  was  in  the  most  advantageous  position  of  the  four  experi- 
mental groups:     these  classes  were  taught  with  the  revised  material  which 
contained  more  exercises  in  computation,    and  they   were  tested  after 


We  are  indebted  to  Dr.    Frank  "Watson  of  the  Office  of  Instructional  Tele- 
vision,   University  of  Illinois,    for  making  available  to  us  his  program  for 
carrying  out  this  test,    and  to  Mr.   James  Hennes  for  its  actual  execution 
on  the  IBM  7090  computer. 


-14- 


Table  6.      Comparison  of  experimental  groups  E^,    E_,    E.,    E-, 
each  with  control  sample,    on  Coop  Algebra  means 


(a)      Group  E^  vs.    Group  C  ^^^^^^ 

Source Adjusted  S.  S.  n.  d.  f .  Adjusted  Mean-Square      F-ratio 

Between-groups           1103.91  1                           1103.91                      17.57 

Within-groups            49639.25  790                              62.83 

Total                     '  791                               --- 


Group  E-,  =    62.41  ^ 

Adjusted  Coop  Algebra  means  P   <    .001 

Group  C^  ,   ,     =    58.  99 
'^      total 


(b)      Group  E-,  vs.    Group  C  .        , 

Source Adjusted  S.  S.       n.  d.  f .       Adjusted  Mean-Square      F-ratio 

* 
Between-groups  150.53  1  150.53  2.53 

Within-groups  56011.35  942  59-46 

Total  943 


* 


Group  E-  =    60.48 

Adjusted  Coop  Algebra  means  not  signif. 

G^°^PC  total     =    59-56 
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Table  6.      (continued) 


(c)      Group  E.  vs. 

■   ^^°^P^  total 

Source 

Adjusted  S.  S. 

n.  d.  f .       Adjusi 

ted  Mean- 

•Square      F-ratio 

Betwe  en-groups 

45.21 

1 

45.21 

.67* 

Within-groups 

84399. 96 

1246 

67.74 

Total 

1247 



Adjusted  Coop  Algebra  means 

Group  E . 
^^°^P  <=total 

=    59.07 
=    58.68 

5jC 

not  signif. 

(d)     Group  E^  vs.    Group  C  ^^^^^ 

Difference  between  adjusted  means 

(Group  E_  minus  Group  C  ,  )  3.00 

standard  error  of  difference  .  538 

t   =    5.57  (P  <    .001) 


completion  of  Unit  4.     It  would  have  been  quite  discouraging  for  UICSM  if 
this  group  had  not  excelled  over  the  control  sample;   but  it  did. 

The  success  of  Group  E^  (which  combined  the  disadvantage  of  being 
taught  with  pre-1959  material  and  of  being  tested  promptly  at  the  end  of 
the  school  year),    on  the  other  hand,    is  rather  remarkable.     At  least  part 
of  this  success  must  be  attributed  to  the  group's  exceptional  superiority  in 
general  scholastic  ability--as  indicated  by  its  being  considerably  above  the 
overall  experimental-sample  average  on  DAT-VR,   and  the  highest  of  all 
the  groups  on  DAT-NA  (cf.    Table  2).     But  there  is  also  other  evidence 
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indicating  that  more  than  one  of  the  classes  in  this  group  had  highly  com- 
petent teachers  who  successfully  covered  almost  the  entire  first  course  in 
one  school  year.     This  observation  lends  support  to  the  surmise  that,    given 
these  favorable  conditions,    the  UICSM  material,    even  without  augmentation 
of  computational  drill  exercises,   was  adequate  in  preparing  superior  stu- 
dents to  cope  with  such  conventionally  oriented  achievement  tests  as  the 
Coop  Algebra. 

Experimental  jGroups  vs.    Control  Group  C,.     Prior  to  making  the 
above  comparisons  between  each  experimental  group,    in  turn,   with  the 
entire  control  sannple,   we  undertook  to  compare  some  of  the  experimental 
groups  with  control  group  C..     This  group  has  a  somewhat  higher  mean 
Coop  Algebra  score  (58.77)  than  the  other  control  group  (56.  65),    and  we, 
therefore,   thought  that  comparisons  with  group  C.  would  offer  a  more    strin- 
gent test  than  comparisons  Avith  the  entire  control  sample.     However,    an 
obstacle  was  encountered  to  pursuing  this  plan,   which  we  shall  describe 
below,    and  it  was  therefore  decided  to  use  the  entire  control  sample  as  the 
basis  of  comparisons. 

The  difficulty  with  using  control  group  C,   alone  for 
comparisons  stems  from  the  fact  that  the  regression  plane 
of  Coop  Algebra  on  the  two  DAT  subtests  for  this  group  were 
markedly  non-parallel  to  those  of  most  of  the  experimental 
groups.     Thus,   the  equality- of- slopes  condition,   mentioned 
earlier,   would  not  be  satisfied,    and  the  standard  analysis  of 
covariance  method  for  comparing  adjusted  means  would  not 
be  applicable.     There  are  two  alternative  procedures  that 
can  be  used  in  such  cases,    both  of  which  are  based  on  an  un- 
restricted linear  hypothesis  model.     One  of  these  procedures 
w^as  used  for  the  group  E-  versus  control  sample  comparison, 
cited  earlier.     The  other  is  the  Johnson-Neyman  technique, 
which  enables  us  to  specify  three  regions  in  the  covariate 
space  (in  this  study  the  two-dimensional  space  with  DAT-VR 
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and  DAT~NA  as  the  coordinate  axes),    such  that  the  conclu- 
sion to  be  drawn  depends  on  the  region(s)  in  which  the  cen- 
troids  of  the  two  groups  under  comparison  fall.     If  both 
centroids  fall  in,    say,    Region  A,   we  conclude  that  group  A 
has  a  significantly  higher  adjusted  criterion  mean  than  group 
B;    if  both  centroids  fall  in  Region  B,   the  reverse  is  true; 
and  if  one  or  both  of  the  centroids  fall  in  the  third  region,    the 
difference  is  not  significant. 

Unfortunately,    it  appears  that  the  two  procedures  may 
sometimes  lead  to  conflicting  results,    a  case  at  point  being 
the  following  conclusions,    (1)  and  {2)  below,    for  the  com- 
parison between  experimental  group  E,  and  control  group 

(1)  Separate-group  regressions: 

Difference  between  adjusted  Coop  means         =    6.  92 

(in  favor  of  group  E^) 
Stajidard  error  of  difference  =    2.33 

t   =    2.97     (P    <    .01) 

(2)  Johns  on- Ney  man  technique:     the  C,   group  centroid 
falls  in  the  region  favoring  group  E-,    and  the  E- 
centroid  falls  in  the  region  of  non-significance. 

Thus,    it  seems  that,   when  the  equality-of-slopes  condi- 
tion is  not  satisfied,  we  may  not  be  able  to  draw  unequivocal 
conclusions  as  we  can  when  the  standard  covariance  analysis 
is  applicable.     When  the  entire  control  sample  w^as  considered, 
the  regression  plane  was  not  significantly  non-parallel  to  that 
of  most  of  the  experimental  groups.     The  single  exception 
was  the  case  of  group  E_,   already  discussed,    and  it  should 
be  mentioned  here  that  for  this  comparison  the  Johnson- 
Neyman  technique  led  to  the  same  conclusion  as  that  from 
the  separate-group-regressions  approach. 

Group  Ky  vs.   Group  E.;     Group  E_  vs.   Group  E^..     The  last  two  com- 
parisons to  be  reported  are  those  between  pairs  of  groups  in  the  experimental 
sample  itself.     Specifically,    the  purpose  was  to  compare  the  1958  (pre- 
revision)  and  1959  (post-revision)  classes,   holding  relatively  constant 
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the  amount  of  the  course  covered  as  of  the  time  of  testing.     Reference  to 
Table  1   shows  that  groups  E^  and  E.   (end-of-year  testing)  and  groups 
E     and  E^  (end-of-course  testing)  are  the  appropriate  pairs  to  compare. 
The  results  are  presented  in  Table  8. 

Table  8.      Comparisons  within  the  Experimental  Sample 

(Ninth  Grade  Groups) 

(a)     Group  E,  vs.   Group  E. 

Source Adjusted  S.  S.  n.  d.  f.  Adjusted  Mean-Square      F-ratio 

Between-groups  1012.59  1                    1012.59'                        15.63 

Within-groups  44,581.80  688                         64.80 

Total  689 


* 


Group  E^    =    63.49 
Adjusted  Coop  Algebra  means  ^P    <    .001 

Group  E.    =    60.  20 


(b)      Group  E_  vs.    Group  E, 

Source Adjusted  S.  S.  n.  d.  f.  Adjusted  Mean-Square      F-ratio 

Between-groups                 577.10  1                         577.10                           11.86 

Within-groups             31,524.05  648                           48.65 

Total  649                            


* 


Group  E-    =    63.  21 
Adjusted  Coop  Algebra  means  P    <    .01 

Group  E_    =    65.  75 
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It  was  already  mentioned  above  that  group  E-,  was  an  exceptionally- 
superior  group.     It  is  not  surprising,   therefore,   that  it  excelled  over  its 
1959  counter- part,    group  E^,    despite  the  latter' s  having  been  exposed  to 
a  larger  number  of  drill  exercises. 

On  the  other  hand,    of  the  two  groups  which  were  tested  after  com- 
pletion of  the  course,    group  E^^  (1959)  was  significantly  superior  to  group 
E_   (1958)  in  achievement.     Thus,    we  find  at  least  tentative  evidence  that 
the  1959  revision  was  favorable  to  better  preparing  UICSM  students  to 
achieve  those  skills  that  are  stressed  in  conventional  algebra  courses. 

Summary  of  Results 


Since  many  different  comparisons  were  reported  to  the  foregoing,    it 
may  be  helpful  to  present  an  overview  that  will  enable  the  reader  to  see 
the  results  at  a  glance.     "We  do  this  in  two  ways.     First,    Figure  2  shows, 
in  ascending  order,   the  several  group  nneans  on  Coop  Algebra,   adjusted 
by  a  regression  equation  based  on  the  entire  sample  (experimental  and 
control  combined). 
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Figure  2.      Adjusted  Coop  Algebra  means  for  five  experimental 

and  two  control  groups  (adjustments  based  on  regression 
equation  for  combined  sample). 
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Note  that  these  means  do  not  agree  with  those  reported 
earlier  in  different  group-by-group  comparisons.     This  is  be- 
cause the  earlier  adjustments  were  each  based  on  a  regression 
equation  applicable  to  the  particular  pair  of  groups  being 
compared,   while  the  present  adjustments  are  based  on  the 
total  sample.     These  adjusted  means  do  not  enter  into  signi- 
ficance tests,    but  they  give  a  rough  idea  of  what  the  relative 
standings  of  the  several  groups  might  have  been  if  all  groups 
had  been  comparable  in  terms  of  DAT-VR  and  DAT-NA  means. 


Second,   we  show  in  Table  9  a  list  of  all  the  comparisons  between  var- 
ious groups  that  were  made.     An  inequality  sign  in  this  table  indicates 
whether  the  adjusted  Coop  Algebra  mean  for  the  group  named  on  the  left 
margin  was  significantly  greater  than  (>),    significantly  less  than  (<),    or 
not  significantly  different  from  (  =  )  than  that  for  the  other  group. 
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Table  9.      Comparisons  of  adjusted  group  means  on  Coop  Algebra 
Statistical  test  used:     Standard  Covariance  Analysis 


Groups 
Compared 

Adjusted 
Group  Means 

62.50 
60.45 

F-ratio 
31.19 

P 

Conclusions 

^ total 
total 

<    .001 

E 

total    ^    ^ total 

^1 

^(2. .  •  6) 

67.33 

62.76 

34.79 

<    .001 

^1    ^    ^(2...  6) 

^(2...  6) 
^  total 

61.85 
60.04 

14.90 

<    .001 

^<2 

...6)    ^    ^  total 

^2 

^  total 

62.41 
58.99 

17.57 

<    .001 

^2    >    C  total 

^3 
^ total 

60.48 
59.56 

2.53 

>    .05 

^3    -    ^  total 

^4 

^  total 

59.07 
58.68 

.67 

>    .05 

^4   "    ^  total 

^2 
^4 

63.49 
60.20 

15.63 

<    .001 

^2    >    ^4 

^3 
^5 

63.21 

65.75 

11.86 

<    .01 

E5    >   E3 

•22- 


Table  9.    (continued) 


4 
Statistical  test  used:     Unrestricted  Linear  Hypothesis  Model 


Differences  Conclusions 


Between  Standard  Computer  Johnson- 

Adjusted  Error  of  Program  due        Neynnan    .^ 

Compared  Means  Estimate      t- ratio  P  to  Watson*        Technique 


^1 
^2 


1.  11      ,        2.93  .38       >    .05        Cj    '=   C^ 


J'  3.00  .538  5.57       <.001      E5>C,,^^^,       E5    >   C^^^^, 

^  total  (P    <   .05) 

^3 
^  6.92  2.33  2.97        <    .01       E      >    Cj  E^    =    Cj 

^1  (P    >    .05) 

sic  ^^ 

See  page    13  See  page    16 

4 
See  page    13 
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